Method, device and computer program for providing image search information

ABSTRACT

The present invention relates to a method, device and computer program for providing image search information. The method for an electronic device to provide image search information according to the present invention comprises: a step of generating a query by using an image if a predetermined query generation event occurs while playing the image; a step of transmitting the query to a server; and a step of receiving a search result corresponding to the query from the server and displaying same on a screen, wherein the search result is a search result of at least one object included in the image. A purpose of the invention is to provide a method, device and program for providing image search information, allowing the search of a product similar to a product exposed in an image, without an additional input of a search word.

This application is a U.S. National Stage of PCT/KR2017/009250 filedAug. 24, 2017, which claims the priority benefit of Korean PatentApplication No. 10-2016-0109941, filed on Aug. 29, 2016 in the KoreanIntellectual Property Office.

TECHNICAL FIELD

The present disclosure relates to a method, an apparatus, and a computerprogram for providing image search information through an imagereproduction device, and more particularly to a method, an apparatus,and a computer program for recognizing an object in an image andproviding product information similar thereto without input of aseparate search word.

BACKGROUND ART

Online business transactions such as marketing and trade have beenvitalized against a backdrop of recently expanded wired/wirelessInternet environments. In connection with this, while purchasers browsethrough magazines, blogs, or videos in YouTube through a desktop or amobile terminal connected to the Internet and find enjoyable products,the purchasers then search for product names and purchase the products.Examples of this can be seen in cases where a name of a bag carried by afamous actress in an airport, or a name of a baby product shown in anentertainment show attains a high rank in a real-time search listing ofa portal site. However, at this time, a user should open a separatewebpage to conduct a search and search for a product name and a store,and if the user does not know the name, the user cannot easily searchfor the product, which inconveniences the user.

Meanwhile, sellers frequently sponsor video content such as soap operasor movies to promote products. However, even though the sellers pay anexpense for promotion, product names may not be directly exposed due toan issue of indirect advertisements in most cases.

Accordingly, these days, sellers seek various promotion methods tocreate video content including video frames tagged with productinformation for promotion to allow users to see the product informationshown in a video while the video is reproduced. However, according tosuch methods, the user may identify only limited information provided bythe seller (product information tagged to content by the seller) andthus, if a product which the user wants in the video is not a sponsoredproduct, there is still difficulty in searching for information on theproduct. Further, if the product shown in the video is too expensive topurchase, consumers may want to search for similar products which arecheaper. However, it is not easy to specify a name of the product whichthe user has identified in the video and it is difficult to search forsimilar products.

DETAILED DESCRIPTION OF THE INVENTION Technical Problem

The present disclosure has been made to solve the problems and an aspectthereof is to provide a method, an apparatus, and a program forproviding video search information which can search for products similarto a product shown in a video without any search words.

Another aspect of the present disclosure is to provide a method and anapparatus for providing video search information which can reduce a costspent on product advertisements and more effectively promote products byexposing product information while a video is reproduced withoutseparately manufacturing content tagged with product information.

Technical Solution

In accordance with an aspect of the present disclosure, a method ofproviding video search information by an electronic device is provided.The method includes: when a preset query generation event is made duringreproduction of a video, generating a query using the video;transmitting the query to a server; and receiving a search resultcorresponding to the query from the server and displaying the searchresult on a screen, wherein the search result is a search resultincluding product information or advertisement information of one ormore objects included in the video.

In accordance with another aspect of the present disclosure, anelectronic device is provided. The electronic device includes: acontroller configured to generate a query using a video when a presetquery generation event is made during reproduction of a video; acommunication unit configured to transmit the query to a server andreceive a search result corresponding to the query from the server; anda display unit configured to display the received search result on ascreen, wherein the search result is a search result of a predeterminedobject included in the video.

In accordance with another aspect of the present disclosure, a method ofproviding video search information to a client by a server is provided.The method includes: generating a product database including imagefeature information of a predetermined product and a label correspondingto the image feature information of the product; receiving a querygenerated using a video from the client; analyzing the query; when thequery includes an object to be searched for, to which a label isattached, based on a result of the analysis of the query, searching forproduct information corresponding to the label attached to the object tobe searched for or image feature information of the object to besearched for in the product database; and transmitting the search resultto the client.

In accordance with another aspect of the present disclosure, a serverfor providing video search information to a client is provided. Theserver includes: a database unit configured to generate and store aproduct database including image feature information of a predeterminedproduct and a label corresponding to the image feature information ofthe product; a communication unit configured to receive a querygenerated using a video from the client and transmit a search result ofthe query to the client; and a controller configured to analyze thereceived query, search for product information corresponding to a labelof an object included in the query or image feature information of theobject in the database unit, and provide the product information as thesearch result.

Advantageous Effects

According to the present disclosure, it is possible to search forproducts similar to a product shown in a video without any search words.

According to the present disclosure, it is possible to reduce a costspent on product advertisements and more effectively promote products byexposing product information while a video is reproduced withoutseparately creating content tagged with product information.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual diagram illustrating an embodiment of anelectronic device for providing video search information according tothe present disclosure;

FIG. 2 is a conceptual diagram illustrating an embodiment of a serverfor providing video search information by a client according to thepresent disclosure;

FIG. 3 is a flowchart illustrating an embodiment of a method ofproviding video search information by an electronic device according tothe present disclosure;

FIG. 4 is a flowchart illustrating an embodiment of a method ofgenerating a query according to the present disclosure;

FIG. 5 is a flowchart illustrating an embodiment of a method ofextracting an object according to the present disclosure;

FIG. 6 is a flowchart illustrating an embodiment of a method ofextracting an object according to the present disclosure;

FIG. 7 is a flowchart illustrating an embodiment of a method ofproviding video search information by a server according to the presentdisclosure; and

FIG. 8 illustrates an embodiment in which an electronic device providesvideo search information according to the present disclosure.

MODE FOR CARRYING OUT THE INVENTION

The aforementioned objectives, features, and advantages will bedescribed in detail with reference to the accompanying drawings and,accordingly, the technical idea of the present disclosure can be easilyimplemented by those skilled in the art. In the description of thepresent disclosure, when it is determined that a specific description ofknown technologies related to the present disclosure obscures thesubstance of the present disclosure, the detailed description thereofwill be omitted. Hereinafter, exemplary embodiments of the presentdisclosure will be described in detail with reference to theaccompanying drawings. In the drawings, the same reference numeral isused to indicate the same or similar elements, and all combinationsdescribed in the specification and claims may be made in a predeterminedway. Unless defined otherwise, it may be understood that the singularform may imply one or more and the singular expression includes theplural expression.

In the specification, an “object” may be understood as a thing or animage of the thing which can be recognized as a product, and may be animage of an unspecific thing including noise according to an objectextraction condition or an image of a specific thing corresponding to apredefined label. The “object” may correspond to a product in acomprehensive range according to a classification depth of a learnedlabel, but should be understood as a concept which may correspond todetailed characteristics of a product such as a product color, a productpattern, and a product name.

The “label” in the specification is information defining thecharacteristics of the product (or object) as described above and mayinclude, for example, large category information such as women'sclothing, men's clothing, furniture, household items, and kitchensupplies, medium category information such as top, bottom, and(one-piece) dress, and small category information such as color,pattern, brand, and price. This is only an example and the label may beclassified or defined variously according to user settings. The labelaccording to an embodiment of the present disclosure may be learnedthrough machine learning and a label corresponding to image featureinformation may be pre-stored in a server or a client (electronicdevice).

In the specification, an “electronic device” may reproduce a video of adesktop, a smart phone, and a tablet PC and include all types ofelectronic devices capable of displaying search information, and may beunderstood as the same meaning as a device for providing video searchinformation. Further, the electronic device may include a control devicesuch as a set-top box capable of transmitting video content to a displaydevice through transmission/reception of the video content andreproducing a video or displaying search information through the displaydevice.

FIG. 1 is a conceptual diagram illustrating an embodiment of anelectronic device providing video search information according to thepresent disclosure. Referring to FIG. 1, an electronic device 100according to the present disclosure may include a controller 130, acommunication unit 150, and a display unit 170, and may further displayan input unit 190 (not shown). The electronic device 100 according tothe present disclosure may include only the controller 130, thecommunication unit 150, and the input unit 190, and the electronicdevice may transmit video information to an external display devicewithout including the display unit 170.

When a preset query generation event is made during video reproduction,the controller 130 generates a query on the basis of the video.

When a preset time passes or a search event is generated after a pauseevent is generated during the video reproduction, the controller 130 maydetermine that the query generation event is made. For example, when auser pauses the reproduced video and the video is continuously pausedafter a predetermined time (2 seconds) passes, the electronic device 100may determine that the query generation event is made and generate aquery on the basis of a paused video frame.

When the electronic device 100 can be controlled through a voicecommand, the controller 130 may determine that the query generationevent is made if a preset time passes or a search event is generatedafter a voice control mode activation command is received during thevideo reproduction. For example, when the controller 130 is a set-topbox and an activation command of a preset voice recognition applicationsuch as “hi, ***” or “hello, ***” is input through a microphone, thecontroller 130 may activate a voice control mode and control theelectronic device 100 and elements thereof according to a voice commandreceived thereafter. In this case, the controller 130 according to anembodiment of the present disclosure may capture a video of a screenwhile switching to the voice control mode. When a user command includinga search request keyword such as “search” or “find” is input in thevoice control mode through a microphone (when a search event isgenerated), the controller 130 may determine that a query generationevent is made and generate and transmit a query including at least onepiece of object information extracted from a captured screen or a videoframe, an image of one area included in the captured screen, a pluralityof video frames satisfying a preset condition based on the capturedscreen, or one frame of the video. In this example, the voice controlmode may be understood to mean a mode in which the electronic deviceanalyzes a received voice by activating a voice dictation and executesthe analyzed voice command.

The search event is an event which can be recognized as searching for aproduct by the electronic device 100 and may include, for example, anaction of the user clicking or touching a search button displayed afterpausing, an action of the user designating one area of a video frame, oran action of the user clicking or touching one area or a buttonrecognized as an object in a video frame.

Further, when a pause event is generated during the video reproduction,the controller 130 may display the time remaining to generate the queryon the screen, and when there is no user input before the remaining timereaches 0, may determine that the query generation event is made.

When the query generation event is made, the query generated by thecontroller 130 may include at least one of a paused video frame, animage of one area included in a video frame, a plurality of video framessatisfying a preset condition based on a paused screen, or objectinformation extracted from one video frame included in a video. Theobject information may include at least one piece of image featureinformation extracted from a video, image feature information extractedfrom a predetermined object area, learned image feature informationcorresponding to a specific object, or label information.

When the input unit 190 includes a module capable of receiving a voicecommand such as a microphone, the search event may be a user commandincluding a search request keyword as described above in the embodimentor an input thereof.

In addition, when the communication unit 150 includes a communicationmodule capable of receiving a user command input through an externalinput device such as a Bluetooth reception device or an infraredreception device, a user search command received through a remotecontrol or a terminal application may be the search event. This isbecause the electronic device 100 may recognize reception of a searchcommand through the input device as a search for a product.

The image feature information is information locally or globallyextracted from a video, a video frame, or an image of one area includedin the video. The image feature information may be used for machinelearning to generate a label. When there is a pre-learned label,predetermined image feature information may correspond to a specificlabel or a specific object.

The image feature information may include vector information(descriptor) of a whole image, an object area, or an image (imagesegment or patch) in a preset range located near a feature point withinthe image, and the vector information may include information on abrightness change, a color, a pattern (frequency or distribution of avertical edge and a horizontal edge), a slope of an image segment, and abrightness change larger than or equal to a preset threshold. Further,the image feature information may have not only a unique range value foreach object but also a unique range value for each label according to alearning model.

The image of one area included in the video frame may be designated as apolygon B as illustrated in FIG. 8. When the pause event is generated,the controller 130 may provide one area including A recognized as anobject as a guide for user convenience, and at this time the user mayfinally select an area included in a query by controlling the size orshape of the polygon provided as the guide.

Further, when the paused video frame is a blurred frame, it is difficultto detect an object, or in order to increase object extraction accuracyin the server, the controller 130 may insert one or more video framessatisfying a preset condition based on the paused video frame into thequery.

The preset condition to select one or more video frames included in thequery may be configured using a time, a number of frames, the presenceor absence of subtitles, and/or a degree of definition. For example, thecontroller 130 may generate video frames within 1 second before/afterthe paused video frame as one query according to the preset condition.Further, the controller 130 may generate three video frames as one queryby adding video frames before/after the paused video frame one by one.According to another embodiment, when the degree of definition of thepaused video frame is low, the controller 130 may search for a videoframe having a high degree of definition in the video framesbefore/after the paused video frame and insert the found video frameinto the query. In another example, when subtitles are included in thepaused video, the controller 130 may insert only a video frame fromwhich the subtitles are removed into the query.

The controller 130 may include an object extractor 135 for extracting anobject from the video frame and a query generator 137 for generating aquery including an object.

The object extractor 135 may extract one area including learned imagefeature information corresponding to a specific object as an objectthrough a pre-stored leaning model and attach a label corresponding tothe image feature information to the corresponding object. The object inthe present embodiment may be an object which can be defined as aspecific object and may be defined variously according to a label orrange thereof.

The image feature information learned to correspond to the specificobject using the pre-stored learning model may be a global descriptor ora local descriptor according to settings thereof. That is, the objectextractor 135 may extract an object including image feature informationlearned to correspond to a vehicle from the video frame or an objectincluding image feature information learned to correspond to clothing,and may not only identify which type of object the corresponding objectis but also classify characteristics of the object (product type,product brand, color, material, form, pattern, partial designdifference, and feelings which can be defined by humans—cute, chic,vintage, and the like—). Accordingly, it is possible to label a moredetailed product name or characteristics of the product by using theglobal or local descriptor suitable for purposes.

According to another embodiment, the object extractor 135 may extractimage feature information from the video frame, partition one areawithin the video frame according to a preset condition on the basis ofthe extracted feature information, and configure the partitioned area asan object to be searched for. The object extracted by the electronicdevice according to the present disclosure may be determined as apredetermined “object” regardless of object type and thus may beextracted at a high speed but may include noise. Area partition may beperformed using image feature information within the video frame such asa brightness change point, overall color distribution, edge information,and definition information, and object configuration may be performedusing coordinate information on the video frame. A plurality of objectsmay be included in the video frame in which case the plurality ofobjects are extracted.

When the display unit 170 displays a search result, the controller 130may perform control to preferentially display, on a screen, a searchresult of an object including a label which is the same as or similar toan object which the user has found using a user's search log. This is toincrease a probability of the video search resulting in product purchaseand increase user satisfaction by preferentially exposing the searchresult belonging to a field of interest of the user on the screen.Further, the controller 130 may control the display unit 170 to displaysearch results on the screen in an order of search results having highsimilarity to image feature information of the object included in thequery, thereby increasing search accuracy.

The communication unit 150 transmits a query to a server 200 andreceives a search result corresponding to the query from the server 200.The communication unit 150 may include a wired/wireless communicationmodule for communication with the server 200, for example, a mobilecommunication module supporting GSM, CDMA, WCDMA, HSDPA, and LTE.Further, the communication unit 150 may include a wireless communicationmodule such as WLAN, Wi-Fi, Direct, DLNA, Wibro, and Wimax such as acommunication module for short range communication, for example, acommunication module supporting Bluetooth, RFID, infrared communication,UWB, ZigBee, NFC, Wi-Fi, and Wi-Fi Direct. For reference, if thecommunication unit 150 according to the present disclosure is acommunication module supporting data transmission and reception to andfrom the server 200, the communication unit 150 is not limited by ascheme thereof.

The display unit 170 displays a received search result on the screen,and the search result is a search result of a predetermined objectincluded in the video. The search result may include product informationor advertisement information found through the query. The display unitmay display (output) information processed by the electronic device 100and may display an execution screen information of an applicationexecuted in the electronic device 100 or User Interface (UI) or GraphicUser Interface (GUI) information according to the execution screeninformation.

The display unit 170 may include a display module, and the displaymodule may include a touch screen having a mutual layer structure with atouch sensor or integrated with the touch sensor. The touch screen mayfunction as the input unit 190 (not shown) for providing an inputinterface between the electronic device 100 and the user and alsoprovide an interface for displaying information between the electronicdevice 100 and the user. Further, the display module may be astereoscopic display module for displaying a stereoscopic video. Thestereoscopic display may use a three-dimensional display scheme such asa stereoscopic scheme (glasses type), an auto stereoscopic scheme(non-glasses type), and a projection scheme (holographic type). Thedisplay unit 170 according to the present disclosure is not limited bythe type of display module and may include any display module capable ofdisplaying a moving picture by all types of video reproduction programswhich can reproduce videos and displaying a search result of a query.

The display unit 170 does not mean only an element coupled to theelectronic device 100 in hardware but may be a separate display moduleconnected to the controller 130 wirelessly or through a wire to transmitand receive a video and a control signal.

FIG. 2 is a conceptual diagram illustrating an embodiment of a serverfor providing video search information to a client according to thepresent disclosure.

Referring to FIG. 2, the server 200 according to an embodiment of thepresent disclosure may include a controller 230, a communication unit250, and a database unit 270.

The controller 230 analyzes a query received from the client 100,searches for product information corresponding to a label of an objectincluded in the query or image feature information of the object in thedatabase unit 270, and provides the found product information as asearch result.

When the query includes image feature information classified as noise orlabel information learned as noise, the controller 230 may exclude thecorresponding query or the object from a target to be searched for.

Further, when the query includes a video frame, an image of one areaincluded in the video frame, or a first object to which no label isattached on the basis of the query analysis result, the controller 230may extract image feature information from the video frame, the image,or the first object. The first object is an object to which no label isattached and which is recognized as a thing by the electronic deviceregardless of the product type, and the image may be an image in onearea captured or cropped by the electronic device.

When the video frame, the image, or the first object includes imagefeature information pre-learned as a specific object on the basis of thesearch result of the image feature information, the controller 230 mayconfigure one area including the image feature information in the videoframe, the image, or the first object as an object to be searched forand attach a label corresponding to the image feature information to theobject to be searched for.

The controller 230 searches for a product having the same/similar labelor the same/similar image feature information in the database unit 270on the basis of the label of the object to be searched for or the imagefeature information.

When the query includes the object to be searched for, to which thelabel is attached, on the basis of the query analysis result of thecontroller 230, the controller 230 searches for product informationcorresponding to the label attached to the object to be searched for orimage feature information of the object to be searched for in thedatabase unit 270. The controller 230 controls the communication unit250 to transmit the search result to the client.

When the communication unit 250 receives identification information ofthe user and a search log of the user from the client, the controller230 may provide the user with product information corresponding to alabel included in the search log through the communication unit 250 onthe basis of the identification information and the search log receivedfrom the client. More specifically, the controller 230 may assign higherimportance to the product information corresponding to the labelincluded in the search log such that the product information is morepreferentially exposed to the client.

The communication unit 250 receives the query generated using the videofrom the client and transmits the search result of the query to theclient 100. Further, the communication unit 250 may receive both theidentification information of the user and the search log of the userfrom the client. Since an example of a communication module included inthe communication unit 250 is the same as the communication moduleincluded in the communication unit 150, a description thereof will beomitted.

The database unit 170 generates and stores a product database includingimage feature information of a predetermined product and a labelcorresponding to the product image feature information.

Hereinafter, an embodiment of a method of providing video searchinformation according to the present disclosure will be described withreference to FIGS. 3 to 6.

First, referring to FIG. 3, according to a method of providing videosearch information according to the present disclosure, when a presetquery generation event is made during video reproduction in S100, theelectronic device generates a query using the video in S200 andtransmits the query to the server in S300. The query generated in S200may include object information extracted from at least one piece ofobject information extracted from a video frame (a video frame which isbeing reproduced or a paused video frame) displayed on the screen, animage in an area included in the video frame, a plurality of videoframes which satisfy a preset condition based on the reproduced orpaused screen, or an image frame included in the video. The objectinformation may include at least one piece of image feature informationextracted from the video, image feature information extracted from apredetermined object area, image feature information learned tocorrespond to a specific object, or label information.

When a preset time passes or a search event is generated after a pauseevent is generated during video reproduction, the electronic deviceaccording to the present disclosure may determine that the querygeneration event is generated in S100.

Further, when the pause event is generated during video reproduction,the electronic device according to the present disclosure may displaythe time remaining to generate the query on the screen, and when thereis no user input before the remaining time reaches 0, may determine thatthe query generation event is made in S100.

The query generation event may vary depending on settings. When theelectronic device includes an input module such as a microphone, if theelectronic device may be activated through a voice command or a voicecontrol mode of the electronic device is activated during videoreproduction, the electronic device may automatically capture a videoframe displayed on the screen. When a preset time passes or a searchevent is generated such as reception of a search command in theactivated state, the electronic device may determine that the querygeneration event is made in S100. In this example, the voice controlmode may be understood to mean a mode in which the electronic deviceanalyzes a received voice by activating a voice dictation and executesthe analyzed voice command.

Subsequently, the electronic device receives a search resultcorresponding to the query from the server in S400 and displays thereceived search result on the screen in S500 so as to provide videosearch information. In S500 of displaying the received search result onthe screen, the electronic device may preferentially display the searchresult of an object including a label which is the same as or similar tothe object for which the user has searched using the search log of theuser. This is to increase a probability of the video search resulting inproduct purchase and increase user satisfaction by preferentiallyexposing the search result belonging to a field of interest of the useron the screen of the electronic device. Further, in S500, the electronicdevice may increase accuracy of the search by displaying the searchresult in an order of higher similarity with image feature informationof the object included in the query.

The query generation step S200 using the video may be performed througha process of extracting an object from a video frame in S230 andgenerating a query including the object in S250 as illustrated in FIG.4.

Referring to FIG. 5, in the object extraction step S230, the electronicdevice may extract image feature information from the video frame inS232, partition one area within the video frame according to a presetcondition on the basis of the extracted feature image in S234, andconfigure the partitioned area as an object to be searched for in S236.

Area partition in S234 may be performed using image feature informationwithin the video frame such as a brightness change point, overall colordistribution, edge information, and definition information, and objectconfiguration in S236 may be performed using coordinate information onthe video frame. A plurality of objects may be included in the videoframe in which case the plurality of objects are extracted.

According to another embodiment of the object extraction in S230, theobject extraction in S230 may be performed by extracting an areaincluding image feature information learned to correspond to a specificobject as an object to be searched for on the basis of a pre-storedlearning model in S233 and attaching label corresponding to the imagefeature information to the object to be searched for in S237 asillustrated in FIG. 6.

FIG. 7 is a flowchart illustrating an embodiment of a method ofproviding video search information by a server according to the presentdisclosure. Referring to FIG. 7, the server according to the presentdisclosure generates a product database including image featureinformation of a predetermined product and a label corresponding to theimage feature information of the product.

Upon receiving a query generated using a video from a client in S1000,the server may analyze the query in S2000, determine whether the queryincludes image feature information learned as noise in S2500, and whenthe query includes the image feature information, exclude thecorresponding query from targets to be searched for.

When the query includes a video frame (case 1), an image (not shown) inone area included in the video frame, or a first object to which nolabel is attached (case 2) on the basis of the result of query analysisin S2000, the server searches for image feature information pre-learnedas a specific object in the video frame, the image, or the first objectin S3000. On the basis of the result of the search in S3000, the serverextracts, as an object to be searched for, one area including the imagefeature information from the video frame, the image, or the first objectin S3500. Subsequently, the server attaches a label corresponding to theimage feature information to the object to be searched for in S3700.Further, the server searches for a product having the same/similar labelor the same/similar image feature information in the product database onthe basis of the label of the object to be searched for or the imagefeature information in S4000.

When the query includes the object to be searched for, to which thelabel is attached (case 3), on the basis of the result of query analysisin S2000, the server searches for product information corresponding tothe label attached to the object to be searched for or image featureinformation of the object to be searched for in the product database inS4000. Then, the server transmits the search result to the client inS5000.

Although not illustrated in drawings, the server may receiveidentification information of the user and the search log of the userfrom the client. In this case, in the search result transmission stepS5000, the server may provide the user with product informationcorresponding to the label included in the search log on the basis ofthe identification information and the search log received from theclient. The server may assign higher importance to the productinformation and transmit the product information to the client such thatthe product information is more preferentially exposed to the client.

FIG. 8 illustrates an embodiment of a method of providing video searchinformation by the electronic device according to the presentdisclosure. In FIG. 8, the user reproduces a video through theelectronic device 100. As one example, in a video, a woman is wearing aT-shirt (A) and a user pauses the video intending to search for productinformation of the T-shirt (A). When there is no user input for 2seconds after a pause event is generated, a query generation event ismade by a preset algorithm.

The electronic device 100 generates a query on the basis of the pausedvideo frame according to the made query generation event. The electronicdevice 100 may transmit the video frame itself to the server or maytransmit an image in an area included in the video frame to the server.Further, the electronic device may extract an object from one videoframe included in the video and transmit the extracted objectinformation to the server.

It is assumed that, when the pause event is generated, the electronicdevice is configured to show a rectangular guide (B) to allow the userto designate an area. The user may select the T-shirt (A) by adjustingthe size and shape of the guide (B), or may designate, as an object, atable (C) which the electronic device 100 has not recognized as anobject but in which the user is interested.

In the embodiment in which the electronic device 100 recognizes theaction of clicking or touching the guide area (B) by the user as thequery generation event, the server 200 (not shown) performs a searchusing the guide area image (B). The server 200 receives a queryincluding an image to which no separate label is attached from theelectronic device 100 and identifies whether the image includes imagefeature information pre-learned as a specific object. When the imageincludes image feature information learned as noise, the server 200stops a search for the corresponding image. However, since the imageincludes the T-shirt (A), the image includes image feature informationlearned as the T-shirt.

When the server 200 finds image feature information learned as theT-shirt (a descriptor corresponding to the T-shirt), the server 200configures one area including the corresponding image featureinformation as an object to be searched for. The configured object to besearched for may have the form closest to the T-shirt (A) and may be anarea obtained by excluding an area (background) which is irrelevant tothe product from the initially received image (B).

Subsequently, the server 200 may attach labels such as women's clothing(large category), top (medium category), and blue and T-shirt (smallcategory) to the object to be searched for. Further, the server maysearch for a product corresponding to a blue T-shirt among women's topsin the product database and may perform a search (image search) for aproduct including image feature information similar to the extractedimage feature information in the product database. When the same and/orsimilar products 20A, 10A, 10B, 10C, and 10D are found on the basis ofthe search result, the server 200 transmits information on the sameand/or similar products to the electronic device 100.

The electronic device 100 receives the search result from the server 200and displays the same on the screen. At this time, the electronic device100 may display product information in an area 10 of the screen, an area(B) which the user designates in the video frame, or an area 20 near theobject (A). The product information may be displayed in the area 10 in adescending order or an ascending order according to similarity, interestbased on the search log of the user, price, and popularity. For example,the electronic device 100 may display the product 10A having the highestsimilarity on the top and then display the products 10B and 10C in anorder of high similarity.

Further, the electronic device may display a product name 20B, a productmanufacturer, a price 20C, popularity, a category (label), storeinformation (website, contacts, and location information), and relevantreviews together with the product image 20A and may be configured toexecute another application when corresponding information is clicked ortouched.

The present disclosure can be variously substituted, modified, andchanged by those skilled in the art without departing from the technicalscope of the present disclosure and thus is not limited by theembodiments and the accompanying drawings.

DRAWING LEGEND INSERTION

-   FIG. 1-   130: controller-   135: object extractor-   137: query generator-   150: communication unit-   170: display unit-   FIG. 2-   230: controller-   250: communication unit-   270: database unit-   FIG. 3-   Start-   S100: is query generation event made?-   Yes-   No-   S200: generate query using video-   S300: transmit query-   S400: display search result corresponding to query on screen-   End-   FIG. 4-   Start-   S230: extract object from video frame-   S250: generate query including object-   End-   FIG. 5-   Start-   S232: extract image feature information from video frame-   S234: partition area within video frame using image feature    information-   S236: configure partitioned area as object-   End-   FIG. 6-   Start-   S233: extract area including image feature information as object-   S237: attach label to object-   End-   FIG. 7-   Start-   S1000: generate and store product database-   S1500: receive query-   S2000: analyze query-   S2500: is there image feature information learned as noise?-   Yes-   No-   : information included in query-   Case1    : case1 video frame-   Case2    : case2 object to which no label is attached-   Case3    : case3 object to which label is attached-   S3000: search for image feature information-   S3300: is there image feature information learned as object-   S3500: extract area including image feature information as object to    be searched for-   S3700: attach label-   S4000: search for product using label or image feature information-   S5000: transmit search result to client-   End

The invention claimed is:
 1. A method of providing video searchinformation by an electronic device, the method comprising: displaying atime remaining to generate a query on a screen when a pause event isdetected during reproduction of a video; and determining that a firstquery generation event is made when there is no user input before theremaining time reaches 0; based on a determination that the first querygeneration event was made during reproduction of the video, generatingthe query using the video; transmitting the query to a server; andreceiving a search result corresponding to the query from the server anddisplaying the search result on the screen, wherein the search result isa search result including product information or advertisementinformation of one or more objects included in the video.
 2. The methodof claim 1, wherein the query includes at least one piece of objectinformation extracted from a video frame displayed on a screen, an imagein one area included in the video frame, a plurality of video frameswhich satisfy a preset condition based on the screen, or one frame ofthe video.
 3. The method of claim 2, wherein the object informationincludes at least one piece of image feature information extracted fromthe video or label information learned to correspond to a specificobject.
 4. The method of claim 1, wherein the generating of the querycomprises determining that a second query generation event is made whena preset time passes or a search event is generated after a pause eventis generated during the reproduction of the video.
 5. The method ofclaim 1, wherein the generating of the query comprises determining thata second query generation event is made when a search event is generatedduring the reproduction of the video.
 6. The method of claim 1, whereinthe generating of the query comprises: capturing the video when a voicecontrol mode activation command is received during the reproduction ofthe video; and generating the query using the captured image when asearch event is generated in a voice control mode.
 7. The method ofclaim 1, wherein the generating of the query comprises: extracting oneor more objects from a video frame displayed on a paused screen; andgenerating a query including the one or more objects.
 8. The method ofclaim 7, wherein the extracting of the objects comprises extracting onearea including image feature information learned to correspond to aspecific object through a pre-stored learning model as an object to besearched for and attaching a label corresponding to the image featureinformation to the object to be searched for.
 9. The method of claim 7,wherein the extracting of the objects comprises: extracting imagefeature information from the video frame; partitioning one area withinthe video frame according to a preset reference using the extractedimage feature information; and configuring the partitioned area as anobject to be searched for.
 10. The method of claim 1, wherein thedisplaying of the search result on the screen comprises preferentiallydisplaying, on the screen, a search result of an object including alabel which is equal or similar to that of an object for which a userhas searched using a search log of the user.
 11. A non-transitorycomputer-readable medium comprising instructions executable by theprocessor to: display a time remaining to generate a query on a screenwhen a pause event is detected during reproduction of a video; determinethat a query generation event is made when there is no user input beforethe remaining time reaches 0; based on a determination that the querygeneration event was made during reproduction of the video, generate thequery using the video; transmit the query to a server; and receive asearch result corresponding to the query from the server and display thesearch result on the screen, wherein the search result is a searchresult including product information or advertisement information of oneor more objects included in the video.